System and method for matching of block and slice histological samples

ABSTRACT

Features are disclosed for imaging block and slice samples using an imaging system. The imaging system can link the images by identifiers associated with the block and slice samples. The imaging system can train a machine learning algorithm based on correctly linked images. In some embodiments, the trained machine learning algorithm may include an image analysis module or a convolutional neural network. The imaging system can use the trained machine learning algorithm in order to determine a confidence score of a match between the block and the slice samples. The trained machine learning algorithm can use features of the block and the slice samples such as shape and tissue morphology to determine whether the samples match. In some embodiments, when the confidence score is below a certain threshold, the imaging system can alert a user that the samples may not match.

RELATED APPLICATION(S)

This application claims the benefit of priority of U.S. ProvisionalPatent Application No. 63/132,345, filed Dec. 30, 2020, entitled SYSTEMAND METHOD FOR MATCHING OF BLOCK AND SLICE HISTOLOGICAL SAMPLES, whichis incorporated herein by reference in its entirety.

TECHNICAL FIELD

This disclosure generally relates to an imaging system, such as forcapturing images of a block and/or a slice of a histological sample.

BACKGROUND

An imaging system can be used to capture an image of a desired scene.Thereafter, the image can be used for a variety of purposes, including,for example, image analysis. For example, an imaging system can capturean image and perform image analysis on the image to determine particularimage characteristics of the image. Examples of imaging systems include,but are not limited to, cameras, scanners, microscopes, mobile devices,tablets, laptops, and/or wearable electronics.

SUMMARY

One aspect of the present disclosure is an image analysis apparatus. Theimage analysis apparatus can include a first imaging device that isconfigured to scan a tissue sample block and generate block image databased on the scanning the tissue sample block. The tissue sample blockmay be configured to be sliced to form one or more slices of the tissuesample block. The image analysis apparatus can further include a secondimaging device that is configured to scan a slice of the one or moreslices of the tissue sample block and generate slice image data based onthe scanning the slice. The image analysis apparatus can further includea computing device that is configured to obtain first block image datafrom the first imaging device and obtain first slice image data from thesecond imaging device. The computing device further is configured tolink the first block image data and the first slice image data together.The computing device further is configured to provide the first blockimage data, the first slice image data, and information indicative ofthe first block image data and the first slice image data beingcorrectly linked together as inputs to a machine learning algorithm. Thecomputing device further is configured to train the machine learningalgorithm using the inputs to the machine learning algorithm. Thecomputing device further is configured to obtain second block image datafrom the first imaging device and second slice image data from thesecond imaging device, the second block image data and the second sliceimage data being linked together. The computing device further isconfigured to, based on the second block image data and the second sliceimage data being linked together, perform image analysis on the secondblock image data and the second slice image data using the machinelearning algorithm. The computing device further is configured todetermine, based on an output of the machine learning algorithm, aconfidence value indicative of whether the second block image data andthe second slice image data are correctly linked together.

In another aspect of the present disclosure, the computing device isfurther configured to obtain a confidence threshold associated with auser. The computing device is further configured to compare theconfidence value and the confidence threshold. The computing device isfurther configured to, based at least in part on the comparing of theconfidence value and the confidence threshold, generate a recommendationfor the user indicative of whether the second block image data and thesecond slice image data are correctly linked together.

In another aspect of the present disclosure, the computing device isfurther configured to perform the image analysis using the machinelearning algorithm by providing the second block image data and thesecond slice image data to a trained convolutional neural network. Thetrained convolutional neural network is configured to perform the imageanalysis.

In another aspect of the present disclosure, the computing device isfurther configured to extract a first plurality of features from theblock image data. The first plurality of features include one or more ofa plurality of filters or a plurality of patterns. The computing deviceis further configured to extract a second plurality of features from theslice image data. The second plurality of features include one or moreof a plurality of filters or a plurality of patterns. Performing theimage analysis using the machine learning algorithm includes comparingthe first plurality of features and the second plurality of features.

In another aspect of the present disclosure, the computing device isconfigured to perform the image analysis by using one or more of animage differencing algorithm, a spatial analysis algorithm, a patternrecognition algorithm, a shape comparison algorithm, a colordistribution algorithm, a blob detection algorithm, a template matchingalgorithm, a SURF feature extraction algorithm, an edge detectionalgorithm, a keypoint matching algorithm, a histogram comparisonalgorithm, or a semantic texton forest algorithm.

In another aspect of the present disclosure, the image analysisapparatus further includes a coverslipper and a stainer. The stainerstains the slice of the tissue sample block to generate a stained slice.The coverslipper generates a slide of the stained slice. The secondimaging device is configured to scan the stained slice.

In another aspect of the present disclosure, the first imaging deviceand the second imaging device can be different imaging devices. Inanother aspect of the present disclosure, the first imaging device andthe second imaging device can be the same imaging device.

In another aspect of the present disclosure, the tissue sample block isassociated with a first identifier and the slice of the tissue sampleblock is associated with a second identifier. Linking the block imagedata and the first slice image data is based at least in part on thefirst identifier corresponding to the second identifier.

In another aspect of the present disclosure, the slice of the tissuesample block and the tissue sample block are each associated with atleast one of a tag, an RFID tag, a Bluetooth tag, an identifier, abarcode, a label, a marker, or a stamp.

In another aspect of the present disclosure, the tissue sample blockincludes one or more of a paraffin embedded tissue sample block, anOCT-embedded tissue sample block, a frozen tissue sample block, or afresh tissue sample block.

In another aspect of the present disclosure, the image analysisapparatus further includes a third imaging device. The third imagingdevice is configured to scan a second slice of the one or more slices ofthe tissue sample block to generate third slice image data. Thecomputing device is further configured to obtain the third slice imagedata from the third imaging device and third block image data from thefirst imaging device. The computing device is further configured to linkthe third block image data and the third slice image data together. Thecomputing device is further configured to provide the third block imagedata, the third slice image data, and information indicative of thethird block image data and the third slice image data being correctlylinked together as further inputs to the machine learning algorithm.

In another aspect of the present disclosure, the image analysisapparatus further includes a microtome configured to slice the tissuesample block to generate the one or more slices of the tissue sampleblock.

In another aspect of the present disclosure, the first imaging device isa microtome. The second imaging device includes one or more of themicrotome, a cover-slipper, a case folder imaging station, a singleslide imaging station, a dedicated low resolution imaging device, or adigital pathology scanner.

In another aspect of the present disclosure, the computing device isfurther configured to transmit the recommendation to a user computingdevice associated with the user.

In another aspect of the present disclosure, the computing device isfurther configured to cause display of the recommendation via the usercomputing device associated with the user.

In another aspect of the present disclosure, the computing device isfurther configured to obtain a response to the recommendation, theresponse including an acceptance of the recommendation. The computingdevice is further configured to adjust the confidence threshold based onthe response corresponding to the acceptance of the recommendation.

In another aspect of the present disclosure, the computing device isfurther configured to obtain a response to the recommendation, theresponse including a rejection of the recommendation. The computingdevice is further configured to adjust the confidence threshold based onthe response corresponding to the rejection of the recommendation.

In another aspect of the present disclosure, the computing device isfurther configured to obtain a response to the recommendation. Thecomputing device is further configured to adjust the machine learningalgorithm based on the response to the recommendation.

In another aspect of the present disclosure, the recommendation includesa similarity score.

In another aspect of the present disclosure, the confidence threshold isassociated with a plurality of users. The computing device furtherdetermines the confidence threshold based on a plurality ofcharacteristics associated with the plurality of users.

In another aspect of the present disclosure, the computing device isconfigured to provide the recommendation via an application programminginterface.

The foregoing summary is illustrative only and is not intended to belimiting. Other aspects, features, and advantages of the systems,devices, and methods and/or other subject matter described in thisapplication will become apparent in the teachings set forth below. Thesummary is provided to introduce a selection of some of the concepts ofthis disclosure. The summary is not intended to identify key oressential features of any subject matter described herein.

BRIEF DESCRIPTION OF THE DRAWINGS

Various examples are depicted in the accompanying drawings forillustrative purposes, and should in no way be interpreted as limitingthe scope of the examples. Various features of different disclosedexamples can be combined to form additional examples, which are part ofthis disclosure.

FIG. 1 depicts a schematic diagram of an example networked environmentaccording to some embodiments.

FIG. 2 depicts an example workflow for generating image data from atissue sample block according to some embodiments.

FIG. 3A illustrates an example tissue block sample according to someembodiments.

FIG. 3B illustrates an example tissue block sample and an example tissueslice sample affixed to a slide according to some embodiments.

FIG. 4 depicts an imaging system for capturing images of thehistological samples according to some embodiments.

FIG. 5 depicts a schematic diagram of an image analysis module,including multiple layers of the neural network in accordance withaspects of the present disclosure.

FIG. 6 depicts a schematic diagram of an image analysis module,including multiple convolution networks in accordance with aspects ofthe present disclosure.

FIG. 7 is a flowchart of an example routine for performing imageanalysis on image data from a tissue simple block and a slice of thetissue sample block affixed to a slide according to some embodiments.

FIG. 8 shows an example computing device that may be used to implementaspects of the present disclosure.

DETAILED DESCRIPTION

Generally described, the present disclosure relates to an imaging systemthat can receive a first image of a histological sample (e.g., a tissueblock) and determine a likelihood of a match between the first image ofthe histological sample and a second image of the histological sample(e.g., a slice of the tissue block affixed to a slide). Based on thelikelihood of the match between the first image and the second image(e.g., a likelihood that the histological sample of the first image anda histological sample of the second image correspond to the samehistological sample), the imaging system can perform various operations,such as generating a recommendation and providing the recommendation toa user via a user computing device.

In order to determine the likelihood of the match between the firstimage and the second image, the imaging system can implement an imageanalysis module (e.g., a convolutional neural network, a machinelearning algorithm, etc.) that analyzes each image. As described herein,the use of an image analysis module within such an imaging system canincrease the accuracy of the imaging process. For example, the imagingsystem may provide a more accurate indication of a match between thefirst image and the second image. By using the image analysis module,the imaging system can efficiently and accurately determine thelikelihood of a match. Furthermore, use of the image analysis module canreduce the amount of erroneous matches or mismatches and can reduce theamount of matches that are provided for secondary analysis (e.g.,reducing the amount of matches that are provided to a user forverification).

As used herein, the term “imaging system” may refer to any electronicdevice or component(s) capable of performing an imaging process. Forexample, an “imaging system” may comprise a scanner, a camera, etc. Insome embodiments, the imaging system may not perform the imaging and,instead, may receive the image data and perform image analysis on theimage data.

As described herein, an imaging system can be used to perform imageanalysis on received image data (e.g., image data corresponding tohistological samples). The imaging system can obtain (e.g., via imagingperformed by the imaging system or via imaging performed by an imagingdevice) image data of a first histological sample and image data of asecond histological sample. Each histological sample can be associatedwith a particular tissue block and/or a section of a particular tissueblock, and, to ensure accurate medical diagnoses, it is important toensure that histological sample images corresponding to a particulartissue block are matched with histological sample images correspondingto the same tissue block (e.g., images that correspond to the sametissue block are linked to that tissue block).

Each histological sample can be associated with an identifier and, basedon these identifiers, the imaging system can compare the images of thehistological samples to verify the accuracy of the linking of thehistological samples. The imaging system can implement an image analysismodule in order to compare the images of the histological samples. Theimaging system can feed the images of the histological samples into theimage analysis module to determine the likelihood of a match between theimages of the histological samples. The image analysis module may betrained on a set of predetermined training image data to identifymatches. Based on this training, the image analysis module may determinea likelihood that the images of the histological samples are a match.For example, the image analysis module may determine that it isextremely unlikely that the images of the histological samples are amatch. The imaging system, based on this likelihood, can generate arecommendation and provide this recommendation to a user of the imagingsystem.

In many conventional cases, implementing a generic imaging system toperform the imaging process may not provide satisfactory results inparticular circumstances or for particular users. Such generic imagingsystems may determine that images of histological samples are a matchbased on a user input. For example, the imaging system may receive anindication that two or more images are a match and the imaging systemmay be unable to verify this match. Such a generic imaging system maycause images to be erroneously matched based upon a user input. Forexample, due to user error, the user may erroneously match image data orerroneously determine that image data does not match. As the image datacorresponds to histological samples (e.g., tissue blocks), it can becrucial to determine accurate matches of the histological samples. Anerroneous match between images corresponding to different histologicalsamples and/or a failure to identify images that correspond to the samehistological sample can result in misdiagnosis. Such a misdiagnosis canlead to additional adverse consequences. Further, by requiring a userinput for each pair of images, the imaging process can result inperformance issues. For example, the imaging process for a genericimaging system may be slow, inefficient, and non-effective. Conventionalimaging systems may therefore be inadequate in the aforementionedsituations.

As imaging systems proliferate, the demand for faster and more efficientimage processing and image analysis has also increased. The presentdisclosure provides a system for analyzing image data with significantadvantages over prior implementations. The present disclosure providessystems and methods that enable an increase in the accuracy of anidentification of a match between image data, relative to traditionalimaging systems, without significantly affecting speed or efficiency.These advantages are provided by the embodiments discussed herein, andspecifically by the implementation of an image analysis module toanalyze the image data and determine the likelihood of a match betweenimage data. Further, the use of sample identifiers allows for theverification of a determination by the convolution neural network thatgiven images are a match or are not a match. The use of the imageanalysis module may further allow the imaging system to determinematches between images based on prior determined matches between images,thereby increasing the accuracy and efficiency of the imaging processingaccording to the above methods.

Some aspects of this disclosure relate to training and using an imageanalysis module configured to receive image data of one or morehistological samples for determining a likelihood that the image datacorresponds to additional image data. The image analysis moduledescribed herein can provide improved accuracy for image analysis byusing a trained module (e.g., a machine learning model, a convolutionalneural network) that is trained to recognize similarities between imagedata. An image analysis module that uses such a trained module is ableto provide increased precision and recall without significantlyimpacting computation speeds provided by conventional image analysissystems. In some embodiments, the image analysis may be based onadditional data. For example, the image analysis may be based at leastin part on identifiers associated with the image data. The identifiersmay include tags, identifiers, or other indicators corresponding toparticular image data.

The identifiers can provide identifying information for particular imagedata. For example, the identifiers can identify a source of thehistological sample, a patient associated with the histological sample,a unique code, or any other identifying information. Based on theidentifiers, the image analysis system can determine that first imagedata (e.g., an image associated with a tissue block) should be comparedwith second image data (e.g., an image associated with a slide oftissue). For example, the image analysis system can determine that ifthe first image data and the second image data share the sameidentifier, the first image data and the second image data should becompared. The identifiers may indicate a user identified match betweenimage data and the image analysis system can determine a likelihood thatthis indication is accurate. In some embodiments, the image analysissystem may not use identifiers to identify possible matches. Instead,the image analysis system may compare first image data with a pluralityof image data stored by the image analysis system. The image analysissystem may further use the identifiers to weight matches or mismatchesidentified by the image analysis system.

As described herein, the image analysis module may include any machinelearning model (e.g., a computing system, a computing device, etc.)and/or any convolutional neural network. Further, the image analysismodule can implement one or more image analysis algorithms which mayinclude, for example, an image differencing algorithm, a spatialanalysis algorithm, a pattern recognition algorithm, a shape comparisonalgorithm, a color distribution algorithm, a blob detection algorithm, atemplate matching algorithm, a SURF feature extraction algorithm, anedge detection algorithm, a keypoint matching algorithm, a histogramcomparison algorithm, a semantic texton forest algorithm, and/or anyother type of image analysis algorithm. The image analysis module mayimplement one or more of the algorithms in order to analyze the imagedata.

As described herein, the image analysis system, based on the imageanalysis performed by the image analysis module, can determine an outputindicating the likelihood of a match between first image data and secondimage data. For example, the image analysis system can determine that itis substantially likely that the first image data and the second imagedata correspond to the same histological sample. Further, based on theoutput, the image analysis system can determine a confidence value thatthe first image data and the second image data correspond to the samehistological sample. For example, if the output indicates that the firstimage data and the second image data are likely not a match, the imageanalysis system can provide a low confidence value that the first imagedata and the second image data are a match. Further, if the outputindicates the first image data and the second image data are likely amatch, the image analysis system can provide a higher confidence valuethat the first image data and the second image data are a match. Theconfidence value may correspond to a rating with regards to a confidencescale. For example, the confidence value may be an alphabetical,numerical, alphanumerical, or symbolical rating (e.g., the confidencevalue may be 70%, indicating a 70% confidence in a match of the imagedata).

As described herein, the image analysis system can determine aconfidence threshold for a particular user associated with the imageanalysis system. For example, the user may be a user requesting theanalysis of the image data. Further, the user may provide a confidencethreshold that indicates a desired confidence value for a match. Forexample, the confidence threshold may indicate that a confidence valueover 70% can be identified as a match and a confidence value at 70% orbelow should be verified by the user. The image analysis system canfurther compare the confidence value corresponding to the comparison ofimage data with the confidence threshold. Based on the comparison of theconfidence value and the confidence threshold, the image analysis systemcan determine a recommendation and provide the recommendation to theuser. It is noted that a confidence threshold level of about 70% is forillustrative purposes, and that the confidence threshold level can belower or higher depending on the particular application, patient data,user preferences, etc.

In the following description, various examples will be described. Forpurposes of explanation, specific configurations and details are setforth in order to provide a thorough understanding of the examples.However, it will also be apparent to one skilled in the art that theexamples may be practiced without the specific details. Furthermore,well-known features may be omitted or simplified in order not to obscurethe examples being described.

System Overview

FIG. 1 illustrates an example environment 100 in which a user and/or asystem may implement an image analysis system 104 according to someembodiments. The image analysis system 104 may perform image analysis onreceived image data. The image analysis system 104 can perform imageanalysis in order to determine a likelihood that first image data of thereceived image data matches second image data of the received imagedata, or that the first image data and the second image data arecorrectly linked or otherwise associated with each other (e.g., thefirst image data and the second image data correspond to the same tissuesample block, the first image data and the second image data correspondto the same patient, etc.). Based on the determined likelihood, theimage analysis system 104 can generate a recommendation.

The image analysis system 104 may perform the image analysis using animage analysis module (not shown in FIG. 1 ). The image analysis system104 may receive the image data from an imaging device 102 and transmitthe recommendation to a user computing device 106 for processing.Although some examples herein refer to a specific type of device asbeing the imaging device 102, the image analysis system 104, or the usercomputing device 106, the examples are illustrative only and are notintended to be limiting, required, or exhaustive. The image analysissystem 104 may be any type of computing device (e.g., a server, a node,a router, a network host, etc.). Further, the imaging device 102 may beany type of imaging device (e.g., a camera, a scanner, a mobile device,a laptop, etc.). In some embodiments, the imaging device 102 may includea plurality of imaging devices. Further, the user computing device 106may be any type of computing device (e.g., a mobile device, a laptop,etc.).

The imaging device 102 may capture and/or generate image data foranalysis. The imaging device 102 may include one or more of a lens, animage sensor, a processor, or memory. The imaging device 102 may receivea user interaction. The user interaction may be a request to captureimage data. Based on the user interaction, the imaging device 102 maycapture image data. In some embodiments, the imaging device 102 maycapture image data periodically (e.g., every 10, 20, or 30 minutes). Inother embodiments, the imaging device 102 may determine that an item hasbeen placed in view of the imaging device 102 (e.g., a histologicalsample has been placed on a table and/or platform associated with theimaging device 102) and, based on this determination, capture image datacorresponding to the item. The imaging device 102 may further receiveimage data from additional imaging devices. For example, the imagingdevice 102 may be a node that routes image data from other imagingdevices to the image analysis system 104. In some embodiments, theimaging device 102 may be located within the image analysis system 104.For example, the imaging device 102 may be a component of the imageanalysis system 104. Further, the image analysis system 104 may performan imaging function. In other embodiments, the imaging device 102 andthe image analysis system 104 may be connected (e.g., wirelessly orwired connection). For example, the imaging device 102 and the imageanalysis system 104 may communicate over a network 108. Further, theimaging device 102 and the image analysis system 104 may communicateover a wired connection. In one embodiment, the image analysis system104 may include a docking station that enables the imaging device 102 todock with the image analysis system 104. An electrical contact of theimage analysis system 104 may connect with an electrical contact of theimaging device 102. The image analysis system 104 may be configured todetermine when the imaging device 102 has been connected with the imageanalysis system 104 based at least in part on the electrical contacts ofthe image analysis system 104. In some embodiments, the image analysissystem 104 may use one or more other sensors (e.g., a proximity sensor)to determine that an imaging device 102 has been connected to the imageanalysis system 104. In some embodiments, the image analysis system 104may be connected to (via a wired or a wireless connection) a pluralityof imaging devices.

The image analysis system 104 may include various components forproviding the features described herein. In some embodiments, the imageanalysis system 104 may include one or more image analysis modules toperform the image analysis of the image data received from the imagingdevice 102. The image analysis modules may perform one or more imagingalgorithms using the image data.

The image analysis system 104 may be connected to the user computingdevice 106. The image analysis system 104 may be connected (via awireless or wired connection) to the user computing device 106 toprovide a recommendation for a set of image data. The image analysissystem 104 may transmit the recommendation to the user computing device106 via the network 108. In some embodiments, the image analysis system104 and the user computing device 106 may be configured for connectionsuch that the user computing device 106 can engage and disengage withimage analysis system 104 in order to receive the recommendation. Forexample, the user computing device 106 may engage with the imageanalysis system 104 upon determining that the image analysis system 104has generated a recommendation for the user computing device 106.Further, a particular user computing device 106 may connect to the imageanalysis system 104 based on the image analysis system 104 performingimage analysis on image data that corresponds to the particular usercomputing device 106. For example, a user may be associated with aplurality of histological samples. Upon determining, that a particularhistological sample is associated with a particular user and acorresponding user computing device 106, the image analysis system 104can transmit a recommendation for the histological sample to theparticular user computing device 106. In some embodiments, the usercomputing device 106 may dock with the image analysis system 104 inorder to receive the recommendation.

In some implementations, the imaging device 102, the image analysissystem 104, and/or the user computing device 106 may be in wirelesscommunication. For example, the imaging device 102, the image analysissystem 104, and/or the user computing device 106 may communicate over anetwork 108. The network 108 may include any viable communicationtechnology, such as wired and/or wireless modalities and/ortechnologies. The network may include any combination of Personal AreaNetworks (“PANs”), Local Area Networks (“LANs”), Campus Area Networks(“CANs”), Metropolitan Area Networks (“MANs”), extranets, intranets, theInternet, short-range wireless communication networks (e.g., ZigBee,Bluetooth, etc.), Wide Area Networks (“WANs”)—both centralized and/ordistributed—and/or any combination, permutation, and/or aggregationthereof. The network 108 may include, and/or may or may not have accessto and/or from, the internet. The imaging device 102 and the imageanalysis system 104 may communicate image data. For example, the imagingdevice 102 may communicate image data associated with a histologicalsample to the image analysis system 104 via the network 108 foranalysis. The image analysis system 104 and the user computing device106 may communicate a recommendation corresponding to the image data.For example, the image analysis system 104 may communicate arecommendation indicating a likelihood that first image data matchessecond image data to the user computing device 106. In some embodiments,the imaging device 102 and the image analysis system 104 may communicatevia a first network and the image analysis system 104 and the usercomputing device 106 may communicate via a second network. In otherembodiments, the imaging device 102, the image analysis system 104, andthe user computing device 106 may communicate over the same network.

With reference to an illustrative embodiment, at [A], the imaging device102 can obtain block data. In order to obtain the block data, theimaging device 102 can image (e.g., scan, capture, record, etc.) atissue block. The tissue block may be a histological sample. Forexample, the tissue block may be a block of biological tissue that hasbeen removed and prepared for analysis. As will be discussed in furtherbelow, in order to prepare the tissue block for analysis, varioushistological techniques may be performed on the tissue block. Theimaging device 102 can capture an image of the tissue block and storecorresponding block data in the imaging device 102. The imaging device102 may obtain the block data based on a user interaction. For example,a user may provide an input through a user interface (e.g., a graphicaluser interface (“GUI”)) and request that the imaging device 102 imagethe tissue block. Further, the user can interact with imaging device 102to cause the imaging device 102 to image the tissue block. For example,the user can toggle a switch of the imaging device 102, push a button ofthe imaging device 102, provide a voice command to the imaging device102, or otherwise interact with the imaging device 102 to cause theimaging device 102 to image the tissue block. In some embodiments, theimaging device 102 may image the tissue block based on detecting, by theimaging device 102, that a tissue block has been placed in a viewport ofthe imaging device 102. For example, the imaging device 102 maydetermine that a tissue block has been placed on a viewport of theimaging device 102 and, based on this determination, image the tissueblock.

At [B], the imaging device 102 can obtain slice data. In someembodiments, the imaging device 102 can obtain the slice data and theblock data. In other embodiments, a first imaging device can obtain aslide including the slice and a second imaging device can obtain theblock data. In order to obtain the slice data, the imaging device 102can image (e.g., scan, capture, record, etc.) a slide of the slice ofthe tissue block. The slice of the tissue block may be affixed to theslide. In some embodiments, the imaging device 102 can image the sliceof the tissue block directly. The slice of the tissue block may be aslice of the histological sample. For example, the tissue block may besliced (e.g., sectioned) in order to generate one or more slices of thetissue block. In some embodiments, a portion of the tissue block may besliced to generate a slice of the tissue block such that a first portionof the tissue block corresponds to the tissue block imaged to obtain theblock data and a second portion of the tissue block corresponds to theslice of the tissue block imaged to obtain the slice data. As will bediscussed in further detail below, various histological techniques maybe performed on the tissue block in order to generate the slice of thetissue block and affix the slice to a slide. The imaging device 102 cancapture an image of the slide and store corresponding slice data in theimaging device 102. The imaging device 102 may obtain the slice databased on a user interaction. For example, a user may provide an inputthrough a user interface and request that the imaging device 102 imagethe slide. Further, the user can interact with imaging device 102 tocause the imaging device 102 to image the slide. In some embodiments,the imaging device 102 may image the tissue block based on detecting, bythe imaging device 102, that the tissue block has been sliced or that aslide has been placed in a viewport of the imaging device 102.

At [C], the imaging device 102 can transmit a signal to the imageanalysis system 104 representing the captured image data (e.g., theblock data and the slice data). The imaging device 102 can send thecaptured image data as an electronic signal to the image analysis system104 via the network 108. The signal may include and/or correspond to apixel representation of the block data and/or the slice data. It will beunderstood that the signal can include and/or correspond to more, less,or different image data. For example, the signal may correspond tomultiple slices of a tissue block and may represent a first slice dataand a second slice data. Further, the signal may enable the imageanalysis system 104 to reconstruct the block data and/or the slice data.In some embodiments, the imaging device 102 can transmit a first signalcorresponding to the block data and a second signal corresponding to theslice data. In other embodiments, a first imaging device can transmit asignal corresponding to the block data and a second imaging device cantransmit a signal corresponding to the slice data.

At [D], the image analysis system 104 can perform image analysis on theblock data and the slice data provided by the imaging device 102. Inorder to perform the image analysis, the image analysis system 104 mayutilize one or more image analysis modules that can perform one or moreimage processing functions. For example, the image analysis module mayinclude an imaging algorithm, a machine learning model, a convolutionalneural network, or any other modules for performing the image processingfunctions. Based on performing the image processing functions, the imageanalysis module can determine a likelihood that the block data and theslice data correspond to the same tissue block. For example, an imageprocessing functions may include an edge analysis of the block data andthe slice data and based on the edge analysis, determine whether theblock data and the slice data correspond to the same tissue block. Theimage analysis system 104 can obtain a confidence threshold from theuser computing device 106, the imaging device 102, or any other device.In some embodiments, the image analysis system 104 can determine theconfidence threshold based on a response by the user computing device106 to a particular recommendation. Further, the confidence thresholdmay be specific to a user, a group of users, a type of tissue block, alocation of the tissue block, or any other factor. The image analysissystem 104 can compare the determined confidence threshold with theimage analysis performed by the image analysis module. Based on thiscomparison, the image analysis system 104 can generate a recommendationindicating a recommended action for the user computing device 106 basedon the likelihood that the block data and the slice data correspond tothe same tissue block.

At [E], the image analysis system 104 can transmit a signal to the usercomputing device 106 representing a recommendation indicating thelikelihood that the block data and the slice data correspond to the sametissue block. The image analysis system 104 can send the recommendationas an electrical signal to the user computing device 106 via the network108. The signal may include and/or correspond to a representation of therecommendation. Based on receiving the signal, the user computing device106 can determine the recommendation. In some embodiments, the imageanalysis system 104 may transmit a series of recommendationscorresponding to a group of tissues blocks, slides, and/or slices. Theimage analysis system 104 can include, in the recommendation, arecommended action of a user. For example, the recommendation mayinclude a recommendation for the user to review the tissue block and theslice affixed to the slide. Further, the recommendation may include arecommendation that the user does not need to review the tissue blockand the slice affixed to the slide. The recommendation may furtherinclude a representation of the strength of the recommendation. Forexample, the recommendation may include a qualifier such as apercentage, a ranking, a phrase (e.g., uncertain, likely, unlikely,etc.), etc. that indicates the strength of the recommendation.

Imaging Prepared Blocks and Prepared Slides

FIG. 2 depicts an example workflow 200 for generating image data from atissue sample block according to some embodiments. The example workflow200 illustrates a process for generating prepared blocks and preparedslides including slices from a tissue block and generating pre-processedimages based on the prepared blocks and the prepared slides. The exampleworkflow 200 may be implemented by one or more computing devices. Forexample, the example workflow 200 may be implemented by a microtome, acoverslipper, a stainer, and an imaging device. Each computing devicemay perform a portion of the example workflow. For example, themicrotome may cut the tissue block in order to generate one or moreslices of the tissue block. The coverslipper may create a first slidefor the tissue block and/or a second slide for a slice of the tissueblock, the stainer may stain each slide, and the imaging device mayimage each slide.

A tissue block can be obtained from a patient (e.g., a human, an animal,etc.). The tissue block may correspond to a section of tissue from thepatient. The tissue block may be surgically removed from the patient forfurther analysis. For example, the tissue block may be removed in orderto determine if the tissue block has certain characteristics (e.g., ifthe tissue block is cancerous). In order to generate the prepared blocks202, the tissue block may be prepared using a particular preparationprocess by a tissue preparer. For example, the tissue block may bepreserved and subsequently embedded in a paraffin wax block. Further,the tissue block may be embedded (in a frozen state or a fresh state) ina block. The tissue block may also be embedded using an optimal cuttingtemperature (“OCT”) compound. The preparation process may include one ormore of a paraffin embedding, an OCT-embedding, or any other embeddingof the tissue block. In the example of FIG. 2 , the tissue block isembedded using paraffin embedding. Further, the tissue block is embeddedwithin a paraffin wax block and mounted on a microscopic slide in orderto formulate the prepared block.

The microtome can obtain a slice of the tissue block in order togenerate the prepared slides 204. The microtome can use one or moreblades to slice the tissue block and generate a slice (e.g., a section)of the tissue block. The microtome can further slice the tissue block togenerate a slice with a preferred level of thickness. For example, theslice of the tissue block may be 1 millimeter. The microtome can providethe slice of the tissue block to a coverslipper. The coverslipper canencase the slice of the tissue block in a slide to generate the preparedslides 204. The prepared slides 204 may include the slice mounted in acertain position. Further, in generating the prepared slides 204, astainer may also stain the slice of the tissue block using any stainingprotocol. Further, the stainer may stain the slice of the tissue blockin order to highlight certain portions of the prepared slides 204 (e.g.,an area of interest). In some embodiments, a computing device mayinclude both the coverslipper and the stainer and the slide may bestained as part of the process of generating the slide.

The prepared blocks 202 and the prepared slides 204 may be provided toan imaging device for imaging. In some embodiments, the prepared blocks202 and the prepared slides 204 may be provided to the same imagingdevice. In other embodiments, the prepared blocks 202 and the preparedslides 204 are provided to different imaging devices. The imaging devicecan perform one or more imaging operations on the prepared blocks 202and the prepared slides 204. In some embodiments, a computing device mayinclude one or more of the tissue preparer, the microtome, thecoverslipper, the stainer, and/or the imaging device.

The imaging device can capture an image of the prepared block 202 inorder to generate the block image 206. The block image 206 may be arepresentation of the prepared block 202. For example, the block image206 may be a representation of the prepared block 202 from one direction(e.g., from above). The representation of the prepared block 202 maycorrespond to the same direction as the prepared slides 204 and/or theslice of the tissue block. For example, if the tissue block is sliced ina cross-sectional manner in order to generate the slice of the tissueblock, the block image 206 may correspond to the same cross-sectionalview. In order to generate the block image 206, the prepared block 202may be placed in a cradle of the imaging device and imaged by theimaging device. Further, the block image 206 may include certaincharacteristics. For example, the block image 206 may be a color imagewith a particular resolution level, clarity level, zoom level, or anyother image characteristics.

The imaging device can capture an image of the prepared slides 204 inorder to generate the slide image 208. The imaging device can capture animage of a particular slice of the prepared slides 204. For example, aslide may include any number of slices and the imaging device maycapture an image of a particular slice of the slices. The slide image208 may be a representation of the prepared slides 204. The slide image208 may correspond to a view of the slice according to how the slice ofthe tissue block was generated. For example, if the slice of the tissueblock was generated via a cross-sectional cut of the tissue block, theslide image 208 may correspond to the same cross-sectional view. Inorder to generate the slide image 208, the prepared slides 204 may beplaced in a cradle of the imaging device (e.g., in a viewer of amicroscope) and imaged by the imaging device. Further, the slide image208 may include certain characteristics. For example, the slide image208 may be a color image with a particular resolution level, claritylevel, zoom level, or any other image characteristics.

The imaging device can process the block image 206 in order to generatea pre-processed image 210 and the slide image 208 in order to generatethe pre-processed image 212. The imaging device can perform one or moreimage operations on the block image 206 and the slide image 208 in orderto generate the pre-processed image 210 and the pre-processed image 212.The one or more image operations may include isolating (e.g., focusingon) various features of the pre-processed image 210 and thepre-processed imaged 212. For example, the one or more image operationsmay include isolating the edges of a slice or a tissue block, isolatingareas of interest within a slice or a tissue block, or otherwisemodifying (e.g., transforming) the block image 206 and/or the slideimage 208. In some embodiments, the imaging device can perform the oneor more image operations on one of the block image 206 or the slideimage 208. For example, the imaging may perform the one or more imageoperations on the block image 206. In other embodiments, the imagingdevice can perform first image operations on the block image 206 andsecond image operations on the slide image 208. The imaging device mayprovide the pre-processed image 210 and the pre-processed image 212 tothe image analysis system to determine a likelihood that thepre-processed image 210 and the pre-processed image 212 correspond tothe same tissue block.

Slicing a Tissue Block

FIG. 3A illustrates an example prepared tissue block 300A according tosome embodiments. The prepared tissue block 300A may include a tissueblock 306 that is preserved (e.g., chemically preserved, fixed,supported) in a particular manner. In order to generate the preparedtissue block 300A, the tissue block 306 can be placed in a fixing agent(e.g., a liquid fixing agent). For example, the tissue block 306 can beplaced in a fixative such as formaldehyde solution. The fixing agent canpenetrate the tissue block 306 and preserve the tissue block 306. Thetissue block 306 can subsequently be isolated in order to enable furtherpreservation of the tissue block 306. Further, the tissue block 306 canbe immersed in one or more solutions (e.g., ethanol solutions) in orderto replace water within the tissue block 306 with the one or moresolutions. The tissue block 306 can be immersed in one or moreintermediate solutions. Further, the tissue block 306 can be immersed ina final solution (e.g., a histological wax). For example, thehistological wax may be a purified paraffin wax. After being immersed ina final solution, the tissue block 306 may be formed into a preparedtissue block 300A. For example, the tissue block 306 may be placed intoa mould filled with the histological wax. By placing the tissue block inthe mould, the tissue block 306 may be moulded (e.g., encased) in thefinal solution 304. In order to generate the prepared tissue block 300A,the tissue block 306 in the final solution 304 may be placed on aplatform 302. Therefore, the prepared tissue block 300A may begenerated. It will be understood that the prepared tissue block 300A maybe prepared according to any tissue preparation methods.

FIG. 3B illustrates an example prepared tissue block 300A and an exampleprepared slide 300B with an affixed tissue slice according to someembodiments. The prepared tissue block 300A may include the tissue block306 encased in a final solution 304 and placed on a platform 302. Inorder to generate the prepared slide 300B, the prepared tissue block300A may be sliced by a microtome. The microtome may include one or moreblades to slice the prepared tissue block 300A. The microtome may take across-sectional slice 310 of the prepared tissue block 300A using theone or more blades. The cross-sectional slice 310 of the prepared tissueblock 300A may include a slice 310 (e.g., a section) of the tissue block306 encased in a slice of the final solution 304. In order to preservethe slice 310 of the tissue block 306, the slice 310 of the tissue block306 may be modified (e.g., washed) to remove the final solution 304 fromthe slice 310 of the tissue block 306. For example, the final solution304 may be rinsed and/or isolated from the slice 310 of the tissue block306. Further, the slice 310 of the tissue block 306 may be stained by astainer. In some embodiments, the slice 310 of the tissue block 306 maynot be stained. The slice 310 of the tissue block 306 may subsequentlybe encased in a slide 308 by a coverslipper to generate the preparedslide 300B. The prepared slide 300B may include an identifier 312identifying the tissue block 306 that corresponds to the prepared slide300B. Not shown in FIG. 3B, the prepared tissue block 300A may alsoinclude an identifier that identifies the tissue block 306 thatcorresponds to the prepared tissue block 300A. As the prepared tissueblock 300A and the prepared slide 300B correspond to the same tissueblock 306, the identifier of the prepared tissue block 300A and theidentifier 312 of the prepared slide 300B may identify the same tissueblock 306.

Imaging Devices

FIG. 4 shows an example imaging device 400, according to one embodiment.The imaging device 400 can include an imaging apparatus 402 (e.g., alens and an image sensor) and a platform 404. The imaging device 400 canreceive a prepared tissue block and/or a prepared tissue slide with anaffixed tissue slice via the platform 404. Further, the imaging devicecan use the imaging apparatus 402 to capture image data corresponding tothe prepared block and/or the prepared slide. The imaging device 400 canbe one or more of a camera, a scanner, a medical imaging device, etc.Further, the imaging device 400 can use imaging technologies such asmicroscopy imaging technologies. For example, the imaging technologiesmay include brightfield microscopy, darkfield microscopy, phase contrastmicroscopy, differential interference contrast microscopy, fluorescencemicroscopy, polarizing microscopy, kohler illumination, oil immersionmicroscopy, light microscopy, immunofluorescence microscopy, chromogenicin situ hybridization microscopy, in situ hybridization microscopy,fluorescence in situ hybridization microscopy, or any other type ofimaging technology. For example, the imaging device can be a darkfieldmicroscope, a brightfield microscope, a fluorescent microscope, etc.

The imaging device 400 may receive one or more of the prepared tissueblock and/or the prepared slide and capture corresponding image data. Insome embodiments, the imaging device 400 may capture image datacorresponding to a plurality of prepared tissue slides and/or aplurality of prepared tissue blocks. The imaging device 400 may furthercapture, through the lens of the imaging apparatus 402, using the imagesensor of the imaging apparatus 402, a representation of a preparedtissue slide and/or a prepared tissue block as placed on the platform.Therefore, the imaging device 400 can capture image data in order forthe image analysis system to compare the image data to determine if theimage data corresponds to the same tissue block.

Imaging Algorithms

FIG. 5 depicts a schematic diagram of an image analysis module 500,including multiple layers of a neural network in accordance with aspectsof the present disclosure. The image analysis module 500 may be or maybe implemented by the image analysis system. The image analysis modulecan implement one or more imaging algorithms in order to compare theimage data to determine if the image data corresponds to the same tissueblock. Further, the image analysis module 500 may correspond to one ormore of a machine learning model, a convolutional neural network, etc.In the example of FIG. 4 , the image analysis module 500 corresponds toa convolutional neural network.

The convolutional neural network can include an input layer 502. Theinput layer 502 may be an array of pixel values. For example, the inputlayer may include a 320×320×3 array of pixel values. Each value of theinput layer 502 may correspond to a particular pixel value. Further, theinput layer 502 may obtain the pixel values corresponding to the image.Each input of the input layer 502 may be transformed according to one ormore calculations

Further, the values of the input layer 502 may be provided to a hiddenlayer 504 of the convolutional neural network. In some embodiments, theconvolutional neural network may include one or more hidden layers. Thehidden layer can include a plurality of neurons that each perform acorresponding function. Further, the hidden layer 504 can perform one ormore additional operations on the values of the input layer 502. Forexample, each neuron of the hidden layer 504 can calculate the weightedsum of inputs (e.g., one or more inputs of the input layer 502 may beadded and weighted). By performing the one or more operations, aparticular hidden layer 504 may be configured to produce a particularoutput. For example, a particular hidden layer 504 may be configured toidentify an edge of a tissue sample and/or a block sample. Further, aparticular hidden layer 504 may be configured to identify an edge of atissue sample and/or a block sample and another hidden layer 504 may beconfigured to identify another feature of the tissue sample and/or ablock sample. Therefore, the use of multiple hidden layers can enablethe identification of multiple features of the tissue sample and/or theblock sample. By identifying the multiple features, the convolutionalneural network can provide a more accurate identification of aparticular image. Further, the combination of the multiple hidden layerscan enable the convolutional neural network to identify anddifferentiate particular tissue blocks and/or tissue slices.

The outputs of the one or more hidden layers 504 may be provided to anoutput layer 506 in order to identify (e.g., predict) a tissue blockassociated with the image. The convolutional neural network can furtheridentify a likelihood that the provided image is associated with aparticular tissue block. Further, when a first image data and a secondimage data are provided to the convolutional neural network, theconvolutional neural network can determine the likelihood that the firstimage data and the second image data correspond to the same tissueblock. In some embodiments, the convolutional neural network may includea pooling layer and/or a fully connected layer.

In order to identify the tissue block associated with a particularimage, the image analysis module 500 may be trained to identify thetissue block. By such training, the trained image analysis module 500 istrained to recognize differences in images and/or similarities inimages. Advantageously, the trained image analysis module 500 is able toproduce an indication of a likelihood that particular sets of image datacorrespond to the same scene (e.g., the same tissue block).

Block training data associated with a tissue block may be provided to orotherwise accessed by the image analysis module 500 (e.g., from ascanner, from a data store, from a database, from memory, etc.) fortraining. The predetermined block training data may include tissue blockdata that has previously been identified (e.g., verified to correspondto a particular tissue block). Further, slice training data associatedwith the same training block may be provided to or otherwise accessed bythe image analysis module 500 (e.g., from a scanner, from a data store,from a database, from memory, etc.) for training. The predeterminedslice training data may include tissue slices that previously beenidentified (e.g., verified to correspond to the same tissue block). Thepredetermined slice training data and the predetermined block trainingdata may be linked (e.g., in a data store, in memory, etc.).

Based on the block training data and the slice training data, the imageanalysis module 500 generates a tissue block training data set fortraining. Further, the image analysis module 500 trains using the tissueblock training data set. The image analysis module 500 may be trained toidentify a level of similarity between first image data and second imagedata. The image analysis module 500 may generate an output that includesa representation (e.g., an alphabetical, numerical, alphanumerical, orsymbolical representation) of the similarity between the first imagedata and the second image data.

In some embodiments, training the image analysis module 500 may includetraining a machine learning model, such as a neural network, todetermine relationships between different image data. The resultingtrained machine learning model may include a set of weights or otherparameters, and different subsets of the weights may correspond todifferent input vectors. For example, the weights may be encodedrepresentations of the pixels of the images. Further, the image analysissystem can provide the trained image analysis module 500 for imageprocessing. In some embodiments, the process may be repeated where adifferent image analysis module 500 is generated and trained for adifferent data domain, a different user, etc. For example, a separateimage analysis module 500 may be trained for each data domain of aplurality of data domains within which the image analysis system isconfigured to operate.

Illustratively, the image analysis system may include and implement oneor more imaging algorithms. For example, the one or more imagingalgorithms may include one or more of an image differencing algorithm, aspatial analysis algorithm, a pattern recognition algorithm, a shapecomparison algorithm, a color distribution algorithm, a blob detectionalgorithm, a template matching algorithm, a SURF feature extractionalgorithm, an edge detection algorithm, a keypoint matching algorithm, ahistogram comparison algorithm, or a semantic texton forest algorithm.The image differencing algorithm can identify one or more differencesbetween first image data and second image data. The image differencingalgorithm can identify differences between the first image data and thesecond image data by identifying differences between each pixel of eachimage. The spatial analysis algorithm can identify one or moretopological or spatial differences between the first image data and thesecond image data. The spatial analysis algorithm can identify thetopological or spatial differences by identifying differences in thespatial features associated with the first image data and the secondimage data. The pattern recognition algorithm can identify differencesin patterns of the first image data and the second image data. Thepattern recognition algorithm can identify differences in patterns ofthe first image data and patterns of the second image data. The shapecomparison algorithm can analyze one or more shapes of the first imagedata and one or more shapes of the second image data and determine ifthe shapes match. The shape comparison algorithm can further identifydifferences in the shapes.

The color distribution algorithm may identify differences in thedistribution of colors over the first image data and the second imagedata. The blob detection algorithm may identify regions in the firstimage data that differ in image properties (e.g., brightness, color)from a corresponding region in the second image data. The templatematching algorithm may identify the parts of second image data thatmatch a template (e.g., first image data). The SURF feature extractionalgorithm may extract features from the first image data and the secondimage data and compare the features. The features may be extracted basedat least in part on particular significance of the features. The edgedetection algorithm may identify the boundaries of objects within thefirst image data and the second image data. The boundaries of theobjects within the first image data may be compared with the boundariesof the objects within the second image data. The keypoint matchingalgorithm may extract particular keypoints from the first image data andthe second image data and compare the keypoints to identify differences.The histogram comparison algorithm may identify differences in a colorhistogram associated with the first image data and a color histogramassociated with the second image data. The semantic texton forestsalgorithm may compare semantic representations of the first image dataand the second image data in order to identify differences. It will beunderstood that the image analysis system may implement more, less, ordifferent imaging algorithms. Further, the image analysis system mayimplement any imaging algorithm in order to identify differences betweenthe first image data and the second image data. Based on the identifieddifferences, the image analysis system can determine a likelihood thatthe first image data and the second image data are a match (e.g.,correspond to the same tissue block).

FIG. 6 depicts a schematic diagram of an image analysis module 600,including multiple neural networks in accordance with aspects of thepresent disclosure. The image analysis module 600 may be a Siamesenetwork which can measure the degree of similarity between the featurespace (e.g., embedding) of block image data and the feature space ofslice image data. The image analysis module 600 may include a firstconvolutional neural network 604A for processing the block image data602A and a second convolutional neural network 604B for processing theslice image data 602B. In some embodiments, the first convolutionalneural network 604A and the second convolutional neural network 604B maybe identical twin architectures. The image analysis module 600 mayinclude a third convolutional neural network 606 for processing theoutput of the first convolutional neural network 604A and the output ofthe second convolutional neural network 604B. In some embodiments, oneor more of the convolutional neural networks may be a generativeadversarial network. It will be understood that the image analysismodule 600 can include more, less, or different convolutional neuralnetworks.

The block image data 602A and the slice image data 602B may differ basedon rotation, flipping, color, outline, etc. For example, the slice oftissue affixed to the slide may be rotated or flipped relative to thetissue block. Further, the slice of tissue may be stained with one ormore colors that are different from the colors of the tissue block. Inother examples, the slice of tissue may have a different perimeter sizethan the tissue block. In order to identify a level of similaritybetween the block image data 602A and the slice image data 602B, theimage analysis module 600 can perform image analysis.

The image analysis module 600 can perform a multi-faceted imageanalysis. Further, the image analysis module 600 can divide the imageanalysis into multiple facets. One or more facets of the image analysiscan include preprocessing the block image data 602A and/or the sliceimage data 602B. In some embodiments, the image analysis module 600 mayinclude one or more convolutional neural networks to perform thepreprocessing. The image analysis module 600 can include an imageregistration algorithm to modify (e.g., rotate, flip, etc.) one or moreof the block image data 602A and/or the slice image data 602B. Bymodifying one or more of the block image data 602A and/or the sliceimage data 602B, the image registration algorithm can generate modifiedblock image data and/or modified slice image data that are rotatedand/or flipped in the same manner. The image analysis module 600 caninclude an image segmentation algorithm to extract the tissue outlinesof the block image data 602A and/or the slice image data 602B. In someembodiments, a U-net convolutional neural network can perform the imagesegmentation algorithm.

The first convolutional neural network 604A may learn the feature spaceof the block image data 602A and the second convolutional neural network604B may learn the feature space of the slice image data 602B. The firstconvolutional neural network 604A and the second convolutional neuralnetwork 604B may use one or more imaging algorithms to learn the featurespace. The first convolutional neural network 604A and the secondconvolutional neural network 604B may share one or more weights in orderto learn the feature spaces. In some embodiments, one or more of thefirst convolutional neural network 604A or the second convolutionalneural network 604B may perform at least a portion of the preprocessing.The first convolutional neural network 604A and the second convolutionalneural network 604B may output the feature space to the thirdconvolutional neural network 606 for additional processing. The thirdconvolutional neural network 606 can compare the feature space of theblock image data 602A and the feature space of the slice image data 602Bfor similarities. Based at least in part on this comparison, the thirdconvolutional neural network 606 can determine a likelihood that theslide of the slice of the tissue block corresponds to the tissue block(e.g., whether the slice could have been generated from the tissueblock).

Analyzing the Block Image Data and the Slice Image Data

FIG. 7 shows a method 700 executed by an image analysis system,according to some examples of the disclosed technologies. The imageanalysis system may be similar, for example, to the image analysissystem 104, and may include an image analysis module to perform one ormore image analysis algorithms, a microtome, a coverslipper, a stainer,one or more imaging devices, etc. It will be understood that the method700 may be performed by different devices (e.g., a computing device).The process 700 may begin at block 701. The process 700 may beginautomatically upon receiving image data.

In block 702, the image analysis system obtains first block image datafrom a first imaging device and first slice image data from a secondimaging device. The first imaging device may scan a tissue sample blockand generate block image data based on scanning the tissue sample block.The tissue sample block may further be sliced in order to generate oneor more slices of the tissue sample block. The one or more slices of thetissue sample block may be affixed to one or more slides. The secondimaging device may scan a slide of the one or more slides correspondingto a slice of the one or more slices of the tissue sample block and maygenerate slice image data based on the scanning of the slide. The tissuesample block may include one or more of a paraffin embedded tissuesample block, an OCT-embedded tissue sample block, a frozen tissuesample block, or a fresh tissue sample block. The first block image dataand the first slice image data may correspond to the tissue sampleblock. The image analysis system may include a coverslipper and/or astainer. The stainer may stain the slice of the tissue sample block togenerate a stained slice. The coverslipper may display the stained slicein the slide. In some embodiments, the first imaging device and thesecond imaging device may be different imaging devices. In otherembodiments, the first imaging device and the second imaging device maybe the same imaging device. One or more of the first imaging device orthe second imaging device may be a microtome, a cover-slipper, a casefolder imaging station, a single slide imaging station, a dedicated lowresolution imaging device, or a direct part (“DP”) scanner.

In block 704, the image analysis system links the first block image dataand the first slice image data to the tissue sample block. The imageanalysis system may link the first block image data and the first sliceimage data based on determining the first block image data and the firstslice image data correspond to the tissue sample block. In order todetermine the first block image data and the first slice image datacorrespond to the tissue sample block, one or more of the tissue sampleblock and the slice may be associated with an identifier. The linking ofthe first block image data and the first slice image data may be basedat least in part on determining that an identifier of the tissue sampleblock corresponds to an identifier of the slice. The identifier may beone or more of a tag, a radio frequency identification (“RFID”) tag, aBluetooth tag, an identifier, a barcode, a label, a marker, or a stamp.In some embodiments, the image analysis system may receive an inputindicating that the first block image data and the first slice imagedata are correctly linked together (e.g., from a data store, frommemory, etc.). The image analysis system may link the first block imagedata and the first slice image data and may store informationidentifying that the first block image data and the first slice imagedata (e.g., in a data store, in memory, etc.).

In block 706, the image analysis system provides the first block imagedata, the first slice image data, and information indicative of thefirst block image data and the first slice image data being correctlylinked together as inputs to a machine learning algorithm. In providingthe first block image data, the first slice image data, and informationindicative of the first block image data and the first slice image databeing correctly linked together, the image analysis system can train themachine learning algorithm to generate a trained machine learningalgorithm.

In block 708, the image analysis system trains the machine learningalgorithm using the inputs to the machine learning algorithm. Based onthe image analysis system training the machine learning algorithm, theimage analysis system can generate a trained machine learning algorithm.Further, the training of the machine learning algorithm may be based atleast in part on a plurality of user data. The plurality of user datamay include one or more responses by one or more users to one or morerecommendations. For example, if a particular user rejects arecommendation that links particular block image data and particularslice image data, the image analysis system may not recommend that theblock image data and the slice image data be linked together for asubsequent user. Further, the image analysis system may alter subsequentrecommendations for the user based on the rejection (e.g., the imageanalysis may be less likely to make recommendations for the linking ofblock image data and slice image data). The machine learning algorithmmay include one or more of an image differencing algorithm, a spatialanalysis algorithm, a pattern recognition algorithm, a shape comparisonalgorithm, a color distribution algorithm, a blob detection algorithm, atemplate matching algorithm, a SURF feature extraction algorithm, anedge detection algorithm, a keypoint matching algorithm, a histogramcomparison algorithm, or a semantic texton forest algorithm. In someembodiments, the trained machine learning algorithm may be a trainedconvolutional neural network. The machine learning algorithm may includeextracting a first plurality of features from the second block imagedata and a second plurality of features from the second slice imagedata. The first plurality of features and the second plurality offeatures may each include one or more of a plurality of filters or aplurality of patterns. Further, the machine learning algorithm maycompare the first plurality of features and the second plurality offeatures.

In block 710, the image analysis system obtains second block image datafrom the first imaging device and second slice image data from thesecond imaging device. The second block image data and the second sliceimage data may be linked together

In block 712, the image analysis system performs image analysis on thesecond block image data and the second slice image data using thetrained machine learning algorithm. In some embodiments, the imageanalysis system may perform image analysis on a third slice image dataand the second block image data. In other embodiments, the imageanalysis system may perform image analysis on a third slice image dataand the second slice image data. The image analysis system may performimage analysis on any combination of slice image data and block imagedata.

In block 714, the image analysis system determines a confidence valueindicative of whether the second block image data and the second sliceimage data are correctly linked. In some embodiments, the image analysissystem can obtain a confidence threshold associated with a user andcompare the confidence value with the confidence threshold. Based onthis comparison, the image analysis system may generate a recommendationfor the user. The recommendation may include a similarity score,ranking, etc. Further, the image analysis system may transmit therecommendation to a user computing device (e.g., through an applicationprogramming interface) associated with the user for presentation to theuser. In some embodiments, the image analysis system may cause displayof the recommendation via the user computing device. Further, the imageanalysis system can obtain a response to the recommendation indicatingthat the user accepted the recommendation and adjust the confidencethreshold based on the response. In some embodiments, the response mayindicate that the user rejected the recommendation and the imageanalysis may adjust the confidence threshold based on the response.Further, the image analysis system may adjust the machine learningalgorithm based on the response. In some embodiments, the confidencethreshold can be associated with a plurality of users. Further, theimage analysis system can determine the confidence threshold based on aplurality of characteristics associated with the plurality of users.

FIG. 8 illustrates an example computing system 800 configured to executethe processes and implement the features described above. In someembodiments, the computing system 800 may include: one or more computerprocessors 802, such as physical central processing units (“CPUs”); oneor more network interfaces 804, such as a network interface cards(“NICs”); one or more computer readable medium drives 806, such as ahigh density disk (“HDDs”), solid state drives (“SDDs”), flash drives,and/or other persistent non-transitory computer-readable media; aninput/output device interface 808, such as an input/output (“IO”)interface in communication with one or more microphones; and one or morecomputer readable memories 810, such as random access memory (“RAM”)and/or other volatile non-transitory computer-readable media.

The network interface 804 can provide connectivity to one or morenetworks or computing systems. The computer processor 802 can receiveinformation and instructions from other computing systems or servicesvia the network interface 804. The network interface 804 can also storedata directly to the computer-readable memory 810. The computerprocessor 802 can communicate to and from the computer-readable memory810, execute instructions and process data in the computer readablememory 810, etc.

The computer readable memory 810 may include computer programinstructions that the computer processor 802 executes in order toimplement one or more embodiments. The computer readable memory 810 canstore an operating system 812 that provides computer programinstructions for use by the computer processor 802 in the generaladministration and operation of the computing system 800. The computerreadable memory 810 can further include computer program instructionsand other information for implementing aspects of the presentdisclosure. For example, in one embodiment, the computer readable memory810 may include a machine learning model 814. As another example, thecomputer-readable memory 810 may include image data 816. In someembodiments, multiple computing systems 800 may communicate with eachother via respective network interfaces 804, and can implement multiplesessions each session with a corresponding connection parameter (e.g.,each computing system 800 may execute one or more separate instances ofthe process 700), in parallel (e.g., each computing system 800 mayexecute a portion of a single instance of a process 700), etc.

Certain Terminology

Terms of orientation used herein, such as “top,” “bottom,” “proximal,”“distal,” “longitudinal,” “lateral,” and “end,” are used in the contextof the illustrated example. However, the present disclosure should notbe limited to the illustrated orientation. Indeed, other orientationsare possible and are within the scope of this disclosure. Terms relatingto circular shapes as used herein, such as diameter or radius, should beunderstood not to require perfect circular structures, but rather shouldbe applied to any suitable structure with a cross-sectional region thatcan be measured from side-to-side. Terms relating to shapes generally,such as “circular,” “cylindrical,” “semi-circular,” or“semi-cylindrical” or any related or similar terms, are not required toconform strictly to the mathematical definitions of circles or cylindersor other structures, but can encompass structures that are reasonablyclose approximations.

Conditional language, such as “can,” “could,” “might,” or “may,” unlessspecifically stated otherwise, or otherwise understood within thecontext as used, is generally intended to convey that certain examplesinclude or do not include, certain features, elements, and/or steps.Thus, such conditional language is not generally intended to imply thatfeatures, elements, and/or steps are in any way required for one or moreexamples.

Conjunctive language, such as the phrase “at least one of X, Y, and Z,”unless specifically stated otherwise, is otherwise understood with thecontext as used in general to convey that an item, term, etc. may beeither X, Y, or Z. Thus, such conjunctive language is not generallyintended to imply that certain examples require the presence of at leastone of X, at least one of Y, and at least one of Z.

The terms “approximately,” “about,” and “substantially” as used hereinrepresent an amount close to the stated amount that still performs adesired function or achieves a desired result. For example, in someexamples, as the context may dictate, the terms “approximately,”“about,” and “substantially,” may refer to an amount that is within lessthan or equal to 10% of the stated amount. The term “generally” as usedherein represents a value, amount, or characteristic that predominantlyincludes or tends toward a particular value, amount, or characteristic.As an example, in certain examples, as the context may dictate, the term“generally parallel” can refer to something that departs from exactlyparallel by less than or equal to 20 degrees. All ranges are inclusiveof endpoints.

SUMMARY

Several illustrative examples of comparing a block histological sampleand a slice histological sample have been disclosed. Although thisdisclosure has been described in terms of certain illustrative examplesand uses, other examples and other uses, including examples and useswhich do not provide all of the features and advantages set forthherein, are also within the scope of this disclosure. Components,elements, features, acts, or steps can be arranged or performeddifferently than described and components, elements, features, acts, orsteps can be combined, merged, added, or left out in various examples.All possible combinations and subcombinations of elements and componentsdescribed herein are intended to be included in this disclosure. Nosingle feature or group of features is necessary or indispensable.

Certain features that are described in this disclosure in the context ofseparate implementations can also be implemented in combination in asingle implementation. Conversely, various features that are describedin the context of a single implementation also can be implemented inmultiple implementations separately or in any suitable subcombination.Moreover, although features may be described above as acting in certaincombinations, one or more features from a claimed combination can insome cases be excised from the combination, and the combination may beclaimed as a subcombination or variation of a subcombination.

Any portion of any of the steps, processes, structures, and/or devicesdisclosed or illustrated in one example in this disclosure can becombined or used with (or instead of) any other portion of any of thesteps, processes, structures, and/or devices disclosed or illustrated ina different example or flowchart. The examples described herein are notintended to be discrete and separate from each other. Combinations,variations, and some implementations of the disclosed features arewithin the scope of this disclosure.

While operations may be depicted in the drawings or described in thespecification in a particular order, such operations need not beperformed in the particular order shown or in sequential order, or thatall operations be performed, to achieve desirable results. Otheroperations that are not depicted or described can be incorporated in theexample methods and processes. For example, one or more additionaloperations can be performed before, after, simultaneously, or betweenany of the described operations. Additionally, the operations may berearranged or reordered in some implementations. Also, the separation ofvarious components in the implementations described above should not beunderstood as requiring such separation in all implementations, and itshould be understood that the described components and systems cangenerally be integrated together in a single product or packaged intomultiple products. Additionally, some implementations are within thescope of this disclosure.

Further, while illustrative examples have been described, any exampleshaving equivalent elements, modifications, omissions, and/orcombinations are also within the scope of this disclosure. Moreover,although certain aspects, advantages, and novel features are describedherein, not necessarily all such advantages may be achieved inaccordance with any particular example. For example, some exampleswithin the scope of this disclosure achieve one advantage, or a group ofadvantages, as taught herein without necessarily achieving otheradvantages taught or suggested herein. Further, some examples mayachieve different advantages than those taught or suggested herein.

Some examples have been described in connection with the accompanyingdrawings. The figures are drawn and/or shown to scale, but such scaleshould not be limiting, since dimensions and proportions other than whatare shown are contemplated and are within the scope of the disclosedinvention. Distances, angles, etc. are merely illustrative and do notnecessarily bear an exact relationship to actual dimensions and layoutof the devices illustrated. Components can be added, removed, and/orrearranged. Further, the disclosure herein of any particular feature,aspect, method, property, characteristic, quality, attribute, element,or the like in connection with various examples can be used in all otherexamples set forth herein. Additionally, any methods described hereinmay be practiced using any device suitable for performing the recitedsteps.

For purposes of summarizing the disclosure, certain aspects, advantagesand features of the inventions have been described herein. Not all, orany such advantages are necessarily achieved in accordance with anyparticular example of the inventions disclosed herein. No aspects ofthis disclosure are essential or indispensable. In many examples, thedevices, systems, and methods may be configured differently thanillustrated in the figures or description herein. For example, variousfunctionalities provided by the illustrated modules can be combined,rearranged, added, or deleted. In some implementations, additional ordifferent processors or modules may perform some or all of thefunctionalities described with reference to the examples described andillustrated in the figures. Many implementation variations are possible.Any of the features, structures, steps, or processes disclosed in thisspecification can be included in any example.

In summary, various examples of comparing a block histological sampleand a slice histological sample have been disclosed. This disclosureextends beyond the specifically disclosed examples to other alternativeexamples and/or other uses of the examples, as well as to certainmodifications and equivalents thereof. Moreover, this disclosureexpressly contemplates that various features and aspects of thedisclosed examples can be combined with, or substituted for, oneanother. Accordingly, the scope of this disclosure should not be limitedby the particular disclosed examples described above, but should bedetermined only by a fair reading of the claims. In some embodiments,the image analysis systems disclosed herein can be used to analyzeimages of other samples different than a histological sample.

What is claimed is:
 1. An image analysis apparatus, comprising: a firstscanner configured to scan a tissue sample block and generate blockimage data based on the scanning of the tissue sample block; a secondscanner configured to scan a slice of one or more slices of the tissuesample block and generate slice image data based on the scanning of theslice; and a computing device configured to: obtain the block image datafrom the first scanner and the slice image data from the second scanner,the block image data and the slice image data being linked together;based on the block image data and the slice image data being linkedtogether, perform image analysis on the block image data and the sliceimage data using a machine learning algorithm, wherein the machinelearning algorithm is trained using predetermined training block imagedata and predetermined training slice image data linked with thepredetermined training block image data; and determine, based on anoutput of the machine learning algorithm, a confidence value indicativeof whether the block image data and the slice image data are correctlylinked together.
 2. The image analysis apparatus of claim 1, wherein toscan the tissue sample block, the first scanner is further configured toscan at least one side of the tissue sample block.
 3. The image analysisapparatus of claim 1, wherein the predetermined training block imagedata and the predetermined training slice image data are obtained fromat least one of a data store, the first scanner, or the second scannerand the predetermined training block image data and the predeterminedtraining slice image data are linked together, wherein the machinelearning algorithm is trained using the predetermined training blockimage data, the predetermined training slice image data, and informationindicative of the predetermined training block image data and thepredetermined training slice image data being correctly linked together.4. The image analysis apparatus of claim 1, wherein, to obtain the blockimage data and the slice image data, the computing device is configuredto: obtain the block image data and the slice image data from a datastore.
 5. The image analysis apparatus of claim 4, wherein the blockimage data and the slice image data are linked together in the datastore.
 6. The image analysis apparatus of claim 1, wherein the computingdevice is further configured to: obtain a confidence thresholdassociated with a user; compare the confidence value and the confidencethreshold; and based at least in part on the comparing of the confidencevalue and the confidence threshold, generate a recommendation for theuser indicative of whether the block image data and the slice image dataare correctly linked together.
 7. The image analysis apparatus of claim6, wherein the computing device is further configured to: obtain aresponse to the recommendation; and adjust at least one of the machinelearning algorithm or the confidence threshold based on the response tothe recommendation.
 8. The image analysis apparatus of claim 1, whereinthe computing device is configured to perform the image analysis usingthe machine learning algorithm by providing the block image data and theslice image data to a trained convolutional neural network, wherein thetrained convolutional neural network is configured to perform the imageanalysis.
 9. The image analysis apparatus of claim 1, wherein thecomputing device is further configured to: extract a first plurality offeatures from the block image data, wherein the first plurality offeatures comprise one or more of a first plurality of filters or a firstplurality of patterns; and extract a second plurality of features fromthe slice image data, wherein the second plurality of features compriseone or more of a second plurality of filters or a second plurality ofpatterns, wherein the performing of the image analysis using the machinelearning algorithm comprises comparing the first plurality of featuresand the second plurality of features.
 10. The image analysis apparatusof claim 1, wherein the computing device is configured to perform theimage analysis by using one or more of: an image differencing algorithm;a spatial analysis algorithm; a pattern recognition algorithm; a shapecomparison algorithm; a color distribution algorithm; a blob detectionalgorithm; a template matching algorithm; a SURF feature extractionalgorithm; an edge detection algorithm; a keypoint matching algorithm; ahistogram comparison algorithm; or a semantic texton forest algorithm.11. The image analysis apparatus of claim 1, further comprising: astainer configured to stain the slice of the tissue sample block togenerate a stained slice; and a coverslipper configured to generate aslide of the stained slice, wherein the second scanner is configured toscan the stained slice.
 12. The image analysis apparatus of claim 1,wherein the tissue sample block is associated with a first identifierand the slice of the tissue sample block is associated with a secondidentifier, wherein linking of the block image data and the slice imagedata is based at least in part on the first identifier corresponding tothe second identifier.
 13. A non-transitory computer-readable mediumstoring computer-executable instructions that, when executed by one ormore computing devices, cause the one or more computing devices to:obtain block image data from a first scanner and slice image data from asecond scanner, the block image data and the slice image data beinglinked together, wherein the first scanner is configured to scan atissue sample block and generate the block image data based on thescanning of the tissue sample block, and wherein the second scanner isconfigured to scan a slice of one or more slices of the tissue sampleblock and generate the slice image data based on the scanning of theslice; based on the block image data and the slice image data beinglinked together, perform image analysis on the block image data and theslice image data using a machine learning algorithm, wherein the machinelearning algorithm is trained using predetermined training block imagedata and predetermined training slice image data linked with thepredetermined training block image data; and determine, based on anoutput of the machine learning algorithm, a confidence value indicativeof whether the block image data and the slice image data are correctlylinked together.
 14. The non-transitory computer-readable medium ofclaim 13, wherein the tissue sample block comprises one or more of: aparaffin embedded tissue sample block; a OCT-embedded tissue sampleblock; a frozen tissue sample block; or a fresh tissue sample block. 15.The non-transitory computer-readable medium of claim 13, thenon-transitory computer-readable medium storing furthercomputer-executable instructions that, when executed by the one or morecomputing devices, cause the one or more computing devices to: obtainthe second slice image data from a third scanner and second block imagedata from the first scanner, wherein the third scanner is configured toscan a second slice of the one or more slices of the tissue sample blockand generate second slice image data based on the scanning of the secondslice; link the second block image data and the second slice image datatogether; and provide the second block image data, the second sliceimage data, and information indicative of the second block image dataand the second slice image data being correctly linked together asfurther inputs to the machine learning algorithm.
 16. The non-transitorycomputer-readable medium of claim 13, wherein the first scannercorresponds to a microtome, wherein the second scanner corresponds toone or more of: the microtome; a cover-slipper; a case folder imagingstation; a single slide imaging station; a dedicated low resolutionimaging device; or a digital pathology scanner.
 17. The non-transitorycomputer-readable medium of claim 13, the non-transitorycomputer-readable medium storing further computer-executableinstructions that, when executed by the one or more computing devices,cause the one or more computing devices to: obtain a response to arecommendation; and adjust at least one of the machine learningalgorithm or a confidence threshold based on the response to therecommendation.
 18. A computer-implemented method comprising: obtainingblock image data from a first scanner and slice image data from a secondscanner, the block image data and the slice image data being linkedtogether, wherein the first scanner is configured to scan a tissuesample block and generate the block image data based on the scanning ofthe tissue sample block, and wherein the second scanner is configured toscan a slice of one or more slices of the tissue sample block andgenerate the slice image data based on the scanning of the slice; basedon the block image data and the slice image data being linked together,performing image analysis on the block image data and the slice imagedata using a machine learning algorithm, wherein the machine learningalgorithm is trained using predetermined training block image data andpredetermined training slice image data linked with the predeterminedtraining block image data; and determining, based on an output of themachine learning algorithm, a confidence value indicative of whether theblock image data and the slice image data are correctly linked together.19. The computer-implemented method of claim 18, wherein the tissuesample block comprises one or more of: a paraffin embedded tissue sampleblock; a OCT-embedded tissue sample block; a frozen tissue sample block;or a fresh tissue sample block.
 20. The computer-implemented method ofclaim 18, further comprising: obtaining the second slice image data froma third scanner and second block image data from the first s, whereinthe third scanner is configured to scan a second slice of the one ormore slices of the tissue sample block and generate second slice imagedata based on the scanning of the second slice; linking the second blockimage data and the second slice image data together; and providing thesecond block image data, the second slice image data, and informationindicative of the second block image data and the second slice imagedata being correctly linked together as further inputs to the machinelearning algorithm.